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1 Introduction 

The exact symplectic twist maps of the two-dimensional annulu^ were studied for a 
long time because they represent (via a symplectic change of coordinates) the dynamics 
of the generic symplectic diffeomorphisms of surfaces near their elliptic periodic points 
(see [7]). One motivating example of such a map was introduced by Poincare for the 
study of the restricted 3-Body problem. 

The study of such maps was initiated by G.D. Birkhoff in the 1920s (see |5j). Among 
other beautiful results, he proved that any essential invariant curve by a symplectic 
twist map of the annulus is the graph of a Lipschitz map (an essential curve is a simple 
loop that is not homotopic to a point). He then introduced the notion of instability 
zone. 

Definition. An instability zone of a symplectic twist map / of the annulus is an open 
subset U of the annulus A that is invariant by / and such that: 

- U is homeomorphic to (open) the annulus A; 



^all the definitions are given in subsection 1.2 
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- the closure U ofU in A contains no essential invariant curve that is not contained 
in the boundary dU ; 

- U is a maximal (for the inclusion c) subset of A that satisfies all these properties. 
There are three kinds of instability zones U : 

1. the whole annulus A can be an instability zone; this happens for example for the 
standard map with a large enough parameter (see [H [T71 [T3]): 

2. [/ is a connected component of the complement of an essential invariant curve; 

3. U is bounded; in this case, there exists two Lipschitz functions -0- < ^+ whose 
graphs are invariant and that satisfy: U = {{0,r) G A;^/;_(0) < r < ?/;+(0)}. 

A lot of things are known about the existence of those instability zones. G. D. Birkhoff 
proved in [6j the existence of such instability zones. He even gave the first (and only) 
explicit example of boundary for an instability zone. To visualize his example, imagine 
the time one map T of the rigid pendulum. It is a symplectic twist map with one hy- 
perbolic fixed point and two separatrices connecting this fixed point to itself. Perturb 
T to create one transverse homoclinic intersection at a point of the upper separatrix 
without changing the lower separatrix S. Then S becomes the boundary of an insta- 
bility zone that is above S ('above' is 'inside' in the following picture). 
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In Birkhoff 's example, the boundary of the instabihty zone in non-smooth. Mod- 
ifying the potential in such a way that it has a degenerate minimum, then we obtain 
an similar example for which the boundary of the instability zone is smooth: 




In M. Herman proved that in general, the boundaries of the instability zones 
have an irrational rotation number. Hence Birkhoff 's example is not 'general'. 
Curiously, no other examples of explicit boundaries of instability zone are known. To 
be complete, let us just mention that in [16], J. N. Mather proves that the billiard map 
of a convex billiard whose curvature vanishes at at least one point has an instability 
zone bounded by the boundary of the billiard phase space. Unfortunately, in this case 
(vanishing curvature), the billiard map is not a twist map. . . 

Though we don't know how the 'general' boundaries of the instability zones are, 
we know some facts about what cannot be such a boundary for a sufficiently regular 
symplectic twist map, for example C°°: 

1. it cannot be a curve on which the dynamics is C°° conjugated to a Diophantine 
rotation; indeed, KAM theorems (see [131 El [121 [SSI [TT] for example) implies that 
such a curve is accumulated from below and above by other invariant curves; 
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2. it cannot be a curve on which the dynamics is conjugated to a rational 
rotation; indeed, it is proved in the thesis of R. Douady that in this case you can 
again apply KAM theory. 

Hence a curve that is at the boundary of an instability zone either is not very regular 
or has a rational or Liouville rotation number. We then raise the question: 
Question. Can the boundary of an instability zone with an irrational rotation number 
be non-differentiable? Can it be smooth? 

We will give some answers to these questions in the case of low regularity (C^ or C^). 
At first, we will prove: 

Theorem 1. Let uj € M\Q be an irrational number. In any neighborhood of {9,r) — > 
{0 + r, r) in the topology, there exists a symplectic twist map f of the annulus 
that has an essential invariant curve T such that: 

• /|r is is -conjugated to a Denjoy counter-example and its rotation number is 

• if 'J : T — 7- M is the map whose graph is T, then 7 is at every point except 
along the projection of one wandering orbit (xn), along which 7 has distinct right 
and left derivatives; 

• T is the upper boundary of an instability zone U of f ; 

• there exists two families ofC^ curves 7^, 7^ : M — >• A such that 7^(0) = 7^(0) = Xn 
and: 

(a) f '='ln= ln+1 and / o 7^ = 7^^-^; 

Vy G 7^(M), hm d(r y, /"xq) = anc^Vy € 7o"W, 1™ d{f'^yJ-^xo) = Q; 

n— >+oo n— >-+oo 

(c) 7^(] - 00, 0]) U 7«([0, +oo[) C r and 7^(]0, +oo[) U 7«(] - 00, 0[) C U. 
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Remark. 1) With a slight change in the construction, we can ask that T is the lower 
boundary of the instability zone U; 

2) if we use a Denjoy counter-example with two disjoint orbits of wandering intervals, 
we can do the same construction along two orbits (xn) and (?/„) and the obtain that 
r is the common boundary of two instability zones, the one that is above F and the 
other that is under F; 

3) our counter-example is defined by: fyy{0,r) = {0 + r,r + ip{9 + r)) with Jjip = 0. 
Hence fip{6, r + 1) = fip{0, r) + (0, 1). If a graph T is invariant by f^p, all the translated 
graphs r + (0, A;) with A; G Z are invariant by f^p. This implies that the instability 
zones of ftp are either the whole annulus A or bounded instability zones. Hence lA is 
a bounded instability zone, but the theorem gives us the description of just one con- 
nected component of its boundary. 

4) In [1], we gave an example of a symplectic twist map that has a non differen- 
tiable essential invariant curve with irrational rotation number; here, we improve the 
construction in the following way: 

• using the construction of M. Herman that is given in [11], we manage to improve 
the regularity of our example and obtain a counter-example; 



using a function (/9 : T — )• M whose restriction to a lot of intervals is linear, we 
manage to create a non-trivial stable manifold for the invariant curve; this and 
the fact that the rotation number of f^p restricted to the curve is irrational imply 
that the invariant curve is at the boundary of an instability zone (see subsection 



1.3 for details). 

Indeed, with the notations of this theorem, 7^(M) is a part of of the stable manifold 
of r 

W'{T) = {xek- lim d(/"x,r) = 0} 
and 7"(M) is a part of of the unstable manifold of F 

H^«(r) = {xeK- lim r) = 0}. 



We now explain how the restricted dynamics to any invariant curve of a symplectic 
twist map that has an irrational rotation number can become the dynamics at the 
boundary of an instability zone: 

Theorem 2. Let T he an essential invariant curve of a symplectic twist map f : 
A —7- A whose rotation number is irrational or whose rotation number is rational and 
the dynamics restricted to T is: 

- either conjugated to a rational rotation; 

- or has only hyperbolic periodic points. 
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Then in any neighborhood l/( of f for the C topology, there exists a C symplectic twist 
map g : A ^ A such that: 

1. T is at the boundary of an instability zone of g; 
2- 9\r = f\r- 

We have seen before that such a result is not vahd in C°° topology because of the 
KAM theorems. The tools used to prove theorem [2] are specific to the topology: they 
are the connecting lemma of Hayashi (see [10] ) and more precisely some consequences 
of this connecting lemma that are given in [2]. 

Contrary to Birkhoff 's counter-example or to theorem [T} we have no idea of how 
the stable/unstable manifold of T is in theorem [2| Observe too that for our example 
of theorem [l| we only know a part of the stable/unstable manifold. Hence we raise the 
following question. 

Question. Is it possible to describe (in general or for some specific examples) the 
stable/unstable manifold of the boundary of an instability zone? 

In [14j, P. Le Calvez proves interesting facts concerning the topological structure of 
those sets. 

We observe in subsection 11.31 that the existence of a non-trivial stable manifold for an 
essential invariant curve with an irrational rotation number implies that this curve is 
at the boundary of an instability zone. Hence a related question is: 

Question. Can an irrational essential invariant curve carry a non-uniformly hyper- 
bolic invariant measure? 

Indeed, if this happens, the union of the stable and unstable manifold of the invariant 
measure cannot be contained in the curve and the curve is then at the boundary of an 
instability zone. 

Finally, concerning the first questions that we raised, we obtain an answer just in 
the case of low regularity. Hence the following questions remain open. 

Questions 1 ) Does there exist a smooth (C^ , C^, . . . ) curve with an irrational rotation 
number that is at the boundary of an instability zone for a symplectic twist map 
with k>2? 

2) Does there exist a non curve with an irrational rotation number that is at the 
boundary of an instability zone for a C'^ symplectic twist map with k > 3? 

3) How is a 'typical' boundary of instability zone (is it regular, how is its rotation 
number. . .)? 
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1.1 Structure of the article 

In section [2j we will recall Michel Herman's construction of a symplectic twist map 
that has an essential invariant curve on which the dynamics is Denjoy. In particular, 
we will give some useful estimates in subsection |2.3[ 

In section [3] we will build the counter-example that is descibed in theorem [l] In 
subsection |3.1[ we will construct the homeomorphism that will represent the projected 
dynamics along the invariant curve and we will be more precise about the choice of the 



constants in subsection 3.2 In subsections 3.3 and 3.4, we will prove some estimates. 
Then, in subsection 3.5, we will prove that our modified example is and in section 
3.6 we will determine part of the stable/unstable manifolds of the invariant curve. 
Finally, we will prove theorem [2] in section |4j 

1.2 Notations and definitions 

Notations. • T = M/Z is the circle. 

• A = T X R is the annulus and an element of A is denoted by {0, r). 

• A is endowed with its usual symplectic form, uj = dO Adr and its usual Riemannian 
metric. 

• vr : T X M ^ T is the first projection and vr : ^ M its hft. 

Definition. A diffeomorphism / : A — )• A of the annulus that is isotopic to identity 
is a positive twist map (resp. negative twist map) if, for any given lift / : — >• 
and for every 6 £ M, the maps r i— )• vf o f{9,r) is an increasing (resp decreasing) 
diffeomorphisms. A twist map may be positive or negative. 

Then the maps that we defined just after theorem [T] are positive symplectic twist 
maps. 

Definition. Let 7 : T — )- R be a continuous map. We say that 7 is C-^ at 6* e T if 
there exists a number j'{6) G M such that, for every sequences (6^) and (^^) of points 
of T that converge to 9 such that / then: 

n^oo 9^ - 9i 

where we denote by 9}^ — 9^ the unique number that represents 9^ — 9"^ and that belongs 
to]-i,i]. 

If we assume that 7 is differentiable at every point of T, then this notion of 
coincides with the usual one (the derivative is continuous at the considered point). 
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1.3 Stable manifold of invariant curves 



A consequence of a theorem of J. N. Mather is that if an essential curve T that is 
invariant by a symplectic twist map is at the boundary of an instabiUty zone, then 
W^(r)\r / and this is equivalent too to W^"(r)\r / 0. 

More precisely, in [5], G. D. Birkhoff proved that if U is an instability zone, if C/i is 
a neighborhood of one of its ends (i.e, eventually after compactification, a connected 
component of its boundary) and U2 is a neighborhood of the the other end, then there 
exists an orbit traveling from Ui to U2- This theorem was improved in [181 by J. Mather 
who proved that if Ci, C2 are the ends of U, there exists an orbit whose a-limit set is 
in Ci and cj-limit set is in €2- J- N. Mather used variational arguments and after that, 
P. Le Calvez gave in [T3] a purely topological proof of this result. 

Conversely, let us assume that F is an essential invariant curve that is invariant by a 
symplectic twist / : A — )• A and that W^*(r)\r 7^ 0. The example of the rigid pendulum 
proves that it can happen that F is not at the boundary at an instability zon^ Let 
us assume that /|r has an irrational rotation number or that /|p is conjugated to a 
rational rotation. Suppose that F is not at the boundary of an instability zone. Then 
there exists two sequences of essential invariant curves (F~) and (F^) that are different 
from F such that: 

1. Vn,F+ > F and F" < F; 

2. lim d(F~,F) = and lim d(F+,F) = 0. 

n— ^-oo n— >oo 

Birkhoff 's theorem implies that the curves F~, F+ are equi-Lipschitz and then relatively 
compact for the norm (we speak of the norm of the function whose graph is the 
curve of interest). Let F* be any limit point of one of these two sequences. Then F* 
in an essential invariant curve such that F n F* 7^ 0. Hence /|r* has the same rotation 
number as /|r. 

M. Herman proved in [11] that two curves with the same irrational rotation number 
are equal. Moreover, if the restriction of a symplectic twist map / restricted to an 
essential invariant curve F is conjugated to a rational rotation, all the orbits are 
action minimizing (see e.g. [9]) and a consequence of the results of G. Forni & J. Mather 
contained in [8| (see theorem 13.3) is that when an essential invariant curve is filled by 
periodic orbit, there exists no other minimizing orbit with the same rotation number 
and then no other invariant curve with the same rotation number. Hence any other 
invariant curve and F are disjoint. 

Hence F* = F and the two sequences (F~) and (F+) converge to F. If F^ is the graph 
of 7^, this implies that the sets {{9, r); 7" {6) < r < 7^(6*)} are a base of neighborhood 
of F. Because they are invariant by /, this implies that W'^{'j) = F = VF"(F). We now 
summarize this result and Mather's result: 

^let us recall that we asked that an instability zone is homeomorphic to the open annulus 
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Proposition. Let / : A — J- A 6e a symplectic twist map and let T be an essential 
invariant curve. Then: 

1. if T is at the boundary of an instability zone, then W''{T)\r ^ and PF"(r)\r 7^ 

0; 

2. if W^{T)\r ^ ^ or VF"(r)\r 7^ and if the rotation number of /|p is irrational 
or if it is rational and if /|r is conjugated to a rational rotation, then T is at 
the boudary of an instability zone. 

2 An example due to Michel Herman 

In M. Herman gives an example of a symplectic twist map / : A — t- A that has 
a invariant curve C such that i^^j is C'^-conjugated to a Denjoy counter-example. 
Let us recall his construction. We fix a; G R\Q. 

2.1 Generalized standard map 

The following family of symplectic twist maps was introduced by M. Herman in 
The maps are defined by : 

f^ : T X R T X R; {9,r) ^ {6 + r,r + ip{e + r)). 

where if : T — )• M is a map such that Jj (p{6)d0 = 0. 

As noticed by M. Herman, the main advantage of this map is the following one. Using 
the explicit formula of it is easy to see that the graph of ■0 : T — )• M is invariant by 
ftp if and only if: 

V0 G T, (0 + ip{e),^{e) + ^{d + ij{e))) = {e + ij{d),^{e + vW)). 

If we rewrite this equality and we denote a lift of (7 : T — t- T by ^ : M — t- M, we obtain 
the following criterion for the invariance of the graph of ■(/'• The graph ol tp : T — )• M 
is invariant by f^p if and only if we have: 

• g = Idj + -f/; is an orientation preserving homeomorphism of T; 

• IdK + y = ^{g + g'^). 

In this case, g is none other than the projected dynamics of the restricted to the graph 
of ijj dynamics. In particular, the restricted dynamics is conjugated to g (via the first 
projection). 

Let us give the idea of the construction of M. Herman: he builds a particular Denjoy 
counter-example (7 : T — )• T of rotation number u. Because of Denjoy's theorem, 
such a g cannot be C^. By using very clever estimates, M. Herman proves that f = 
g + g~^ — 2Id is C^. Hence f^, is the wanted counter-example. 
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2.2 Explicit construction of a circle difFeomorphism 



We use the construction that is described in [TT] p. 94, with only a sHght change: we 
define the function r] in such a way that the Denjoy counter-example is linear on some 
small segments. 

Let us recall that we fixed uj G M\Q. Let us fix (5 > and C >> 1. 
We introduce for A; G Z: 



{\k\ + c)(iog{\k\ + c)y+s 

where ac is chosen such that = 1- 

kez 

We use a C°° function rj : M — ^ M such that rj > 0, support(r/) C [\, |], r],,3 si = 1, 

r/(l — t) = ri{t) and ri{t)dt = 1. We define % by : r/fc(^) = ^? Then we have: 



Jq'' r]k{t)dt = ik- Moreover, there exist two constants Ci, C2, that depend only on rj, 
such that : 

C,<\m\<C2; ^<Wk\<^- (1) 

We assume now that C >> 1 is large enough so that: 

4+1 



yk G z. 
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C2 < L 



Then the map % : [0,4] ^ [0,4+i] defined by /ifc(x) = (l + - ij r?fc(t)j 

is a C°° diffeomorphism such that hk(ii^) = ik+i- 

There exists a Cantor subset K C T that has zero Lebesgue measure and that is 
such that the connected components ofT\K, denoted by {Ik)kez, are on T in the same 
order as the sequence {ku) and such that length(/yfc) = £k- 

Let us recall what is the semi-conjugation j : T — t- T of the Denjoy counter-example to 
the rotation Ruj. If x G {kuj; k G Z}, then we define : = dfi{t) where ^ is the 

probability measure /x = ''^^ik^kui, ^kuj being the Dirac mass at kco. Then j : T — )• T 

fcez 

is a continuous map with degree 1 that preserves the order on T and that is such that 
j{Ik) = ku. 

Then there is a diffeomorphism g : T — ?• T that fixes K, is such that K is the 
unique minimal subset for g, has for rotation number p{g) = cj, verifies j o g = ° j- 
If A; G Z, we introduce the notation: g^j^ = g^] then we have: gk{Ik) = -^fc+i- Fol- 
lowing [llj again, we can assume that: (7^ = g'^j^ = ^1 -|- — 1^ ° where 
R-Xkih) = [0,4] and that gk ■ h ^ h+i is defined hy : gk = R\k+^ ohko R-x,,. 
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2.3 Some useful inequalities 

We recall without proof some inequalities that are given in (sometimes we give some 
slight improvement of these inequalities) and that are useful to prove that g + g""^ is 
C^. The constants Ci are independent of k and C >> 1 and the limits are uniform in 
C >> 1. Introduce the notation: = — 1- Then for C >> 1 large enough: 

•f ~>^ K _ , -1 , , -(1 + '^) 

n + C+(n + C)2^(n + C)log(n + C) ^> 

where sup{|e(ibn, k)\]n,C > 1} = c < +oo. If n = 0, the good formula is the formula 
with a +. 

CiKl < \Kk-i - Kk\ =Kk- Kk-i < C2KI (4) 

(|fc| + C)(log(|A;| + - ' - + C)(log(|A:| + C)Y+^ ^ ^ 

lim -7^ = 0. (6) 

We don't recall here how we can deduce the fact that g + g~^ is from these inequal- 
ities, because we will give a very similar proof for the modified example in the next 
section. 

Let us just notice the following fact that is due to our modification of the function 

77: 

Let us now give some estimates that were not given in ^ll]. We introduce the 
notation: := 1 + Kfc + = + |±i. We have: 

ruk+i - 2 - {Kk+i - Kk) = 

hence we deduce from Q that 

|mfc+i - 2| < C2KI and \mk - mk-\-i\ < C2KI. (7) 
Because of ([s]) we deduce that: 

K+i-2|<p^. (8) 
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3 Modification of Michel Herman's example 



3.1 Explicit construction of a circle homeomorphism 

We introduce two new functions 7_, 7+ : R — )• R such that: 

• support (7±) C [0, 1]; 

• = 0; 7+|[o,i] = 0; 

• VtG [|,i[,7_(t) = landVtG]^,|],7+(t) = l; 

• -f±{t)dt = 0. 

Hence these two functions are C°° on R\{2} and discontinuous at the point 
We define a sequence of functions (7^) by: 

7fe(a;) = 7+(7^) if k>l and 7fc(x) = 7_(|^) if A: < 

Let us fix a sequence (a^) of real numbers such that < |ajt| < ^.ji^fcl (where A 
is a constant). Then we define ^/^^ : R — )• R by ipk{x) = Kkr]k{x) + ak'ykix) and a new 
function hk : [Q,^k] — )• R by /ifc(x) = /g^(l + 'ipk{t))dt. If C is large enough {C was 
the constant that is used to define (ik) and then {K^)), then {K^) and (a^) are small 
enough (A is a fixed constant that doesn't depend on C ) and l + ^^k is positive. Hence 
/ifc is a homeomorphism onto [0,^fc_|_i]. 

Let us notice that is differentiable everywhere except at ^ where it has distinct left 
and right derivatives. More precisely: 

1) if A: > 1, then: Vx E [§4, 24], /ifc(t) = and Vx G [^4, §4], /ifc(t) = 

2) if < 0, then: Vx G [§4, i4], /ifc(t) = (^+afc)t-^ and Vx G [^4, i4],/ifc(t) = 

Then with this new functions /i^ we can construct and g exactly as this was done 
in Herman's example. The only difference is that there is a discontinuity of g' at the 
middle of every connected component of the wandering set, the map g being linear on 
a right neighborhood and on a left neighborhood of each such singularity. 
Moreover, h'f, tends to 1 when k tends to ±00. This implies (a precise proof was given 
in [1] ) that g and the curve T are at all the points that are not at the middle of every 
connected component of the wandering set. Observe that the set of discontinuities of 
g corresponds to one orbit. 
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3.2 Choice of a "good" sequence (ak). 

Let us recall that we want that f = g + — 2IdiR is C^. We need to choose carefully 
the sequence (ak) to obtain that. Let us now explain how we choose (ofc), and after 
that we will prove that (p is C"^. 

We begin by choosing two small ai > and ao < such that: 

^ +l + Ki = —^— + l + Ki + ai. 



l + Ko + ao 1 + Kq 

We denote by m this quantity. 

Then we extend this sequence by using the constants := 1 + + ij^k^ i " 

Vfc £ Z\{0},l + Kfc+i + ) =mk+i. (9) 

If we denote by $fc the map :]0, +oo[— ^ M defined by = — j, each is 

increasing and we have: + ^fc) = 1 + -K^fc+i- Because ai > 0, we deduce that 

we can define (an)ra>i by using ^ and that: Vn > l,an > 0. In a similar way, each 
is increasing on ] — oo,mfc[ and oq < 0, hence we can define (a_„)„>i by ^ and 
we have then: Vn > 0, a_n < 0. Similar remarks were done in [1]. 
For this particular choice of (ofc), we can notice that for all A; G Z, /i^ + h'^\ is linear 
in the interval [g^fc, g^fc]- More precisely (we use the fact that the are continuous 
at ^ to determine some constants)): 

• if > 2: if X G [|4, hk{x) + h'^^^{x) = (1 + Kk)x + jipT^^x = nikX and if 

hk{x) + K^ix) = {l + Kk + ak)x -^^ + (x + ^f^^i^) = mfcx; 

• if A; = 1: if x G [^4, §4]: 

hi{x) + /iq ^(x) = (1 + i^i + ai)x - ^ + i^x = mx - = mix - and 

ifxG[|4,i4]: 

/ii(x) + /io^(x) = (1 + Ki)x + {x + ^) =mix-^ (let us notice that 

we change the notation for mi from this point); 

• if /c < 0: if X G [^4) f^fc]) ^fc(a:^) + ^^^^1(2;) = (1 + Kk)x + jqr^^^x = m^x and if 
X G [g^fc ) 2 ^^'] ' 

%(x) + /i-^i(x) = {l + Kk + ak)x + + "'--f-^ ) = mfcx. 

We deduce immediately that the function = g + g^^ — 2IdK is linear on each segment 
Jfc C Ik that is at the middle of Ik and has length ^. In particular, the restriction of 
if to the interior of any interval Ik is C°°. 
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We denote by the C°° function that is equal to 93 on 1^ and equal to every- 
where else. Then: <f = Y1,'-Pk and to prove that is C^, we just have to prove that 
lim ||Z)^(/7fc||(^o = 0. If we want to prove that (pk is close to in topology, we have 



fc— >itoo 

-.2 



to prove that lim sup{||D Vfclico; G Z} = 0. 
C— >+oo 

3.3 Estimation of (a„)„>i 

If we want to have a control on ||D^(/7fc||(^o, we need to have a control of the sequence 
(a/c). We use the following notation: = + oik- 

We have built the sequences (i^fc) and (m^) that depend on a certain constant 
C » 1, we have chosen oi > small and defined: 

Vn > 1, 1 + (3n+i + -1 ^ r, = rrin+i- 

J- + Pn 

We have considered the functions :]0,+oo[— )■ M defined by ^k{t) = rrik — j. 
Then we have: 1 + /3fc+i = ^k+i{^ + Pk)- This function is strictly increasing and 
concave. When mk > 2, ^k has exactly two fixed points ak < 1 < bk and we have: 
bk = \{mk + ^Jml-A) hence (see (8)): 

< 6fc - 1< Ci^M^ < (10) 

n + C 

Let us now compare 1 + = <I>„,-|-i(l + /3„) with 1 + /3„. We fix a constant 

B » 2. There are three cases: 

1. ifmn+i < 2and/3„ < thenl+/3„+i = $„+i(l+/3„) < $„+i(l + ^^^f^) < 

+1+C because $n+i < Id; 

2. ifmn+i > 2and^„ < ^^qif^; then l + /3„+i = $„+i(l + /3„) < $„+i(l + ^^q:^) < 
1 + ^i+rpc because $„+i|[fe„+i,+oo[ < ^[b„+^,+oo[ and 1 + ^^^f^ = 1 + + 
n+i+h - ^^^+1 ^ l^^S^ enough (see (10)); 



if 1 + /3„ > 1 + 

„+f+C • introduce the notation 5„ — 2(„44+c') • '^^^ function 
being concave such that D<I>„+i(l + 6n+i) = Trri-yi, we have: 



(l + 5n)2' 

1 

(1+1. 



1 + /3„+i - «>„+i(l + (5n) < , , (1 + /3„ - (1 + 5„)). 



If m^+i < 2, we have <I>„+i(l + (5„) < 1 + (^n because <I>n+i < Id; if run+i > 2, as 
1 + 5n > bn+i (see point [2]), we have <I>„-|-i(l + 6n) < I + and then: 

^"+^-^'"(rT^^'^" + (riF^"- 
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As 6n < we deduce: 



We deduce for C large enough: 




^ (1 - 3(^ + c + l))^- 

We choose i? > 3. We have then: 

Pn+l < (1 - — ^)/3n = ^t,^, /?n- (H) 

n+C+1 n+C+1 

Let us now prove some estimates for (/3„) (and then (a„)). At first, let us recall 
that f3n > Kn (because we have noticed that a„ > 0). Let us now choose ai > small 
enough such that /3i = ai + < j^', this is possible because Ki <Q (see Now 

we prove by recurrence that: Vn > < ^i^- 
The result is true for n = 1. 

Let us assume that it is true for some n > 1. There are two cases: 



• either /?„ < ^^qrjq:^; then we have proved that: < 



n+l+C 



• or /?„ > then by we have: /3„+i < ^/3„ < = 

Finally, we have proved that: 

Vn>l,K„</3„< 

n + G 

Using Q , we deduce similar estimates for = /?„ — Kn : Vn > 1 , < < 



_ n+c- 

3.4 Estimation of (a-n)n>o 



This time we will use the smallest fixed point = ^(m^ — — 4) of when 

ruk > 2. We have (because of Q, K-n is positive): 

> flfc - 1 > -CiV^^^^ > ^ (12) 

n + O 

We have noticed that: Vn > 0, /3_„ < K^n- Let us now compare 1 + /3-n-i = 
$I„(1 + /3_„) with 1 + (3-n- We fix a constant B » 2. There are three cases: 

1. if m_„ < 2 and /?„ > then 1 + /3_„_i = $1^(1 + /3_„) > <!>zi{l - 

ThT+c)) ^ 1 - ^hTO because $1^ > Id; 
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2. if m_„ > 2 and /?„ > --r^; then 1 + /3_„_i = + > 



, -I , ^ )) > 1 , ^ because <I> , > Idi_n^„ i and 1 , ? , ^ 



1 - n+l+C ~ n+l+C - if is i^^g^ enough (see (12)); 
3. if P-n < — n+l+C ' introduce the notation 7_„ 
being convex such that D{^Zn){l + J-n] 



2(n+1+C) • The function 



c^li(l + 7_„)-(l + /3_„_i) < 



(m_„ - 1 - 7_„)' 



K(l + 7-n)-(l + /3-n)) 



If m^n < 2, we have ^_n(l + 7-n) > 1 + 7-n because <I>„^ > Id; if m_„ > 2, as 
1 + j-n < fl-n (see point 2|, we have ^"1^(1 + 7-n) > 1 + 7-n and then: 



13-n-l > (1 



1 



h-n + 



P-n 



(m-n - 1 - 7-n)^ {m-n - 1 - 7-n) 



Because of (jSj), we have: 

I (m-n - 1 - 7-n) - (1 + 



B 



< 



Co 



2(n + l + C)" - (n + C)2 



(13) 



and then 1 
we deduce: 



— — yi is positive if C is large enough. Because 7-n > , 



/3-n-i > (tt + 



2 2(m_„ - 1 - 7_ 



r)/3- 



and then by ( 13 ): 



/3-n-l > (1 



B 



3(n + 1 + C) 



)(3-n 



If 5 > 3, we obtain: 

/3-n-l > (1 



1 



-)/3-n > r^^^/3-n 



(n + l + C)'"^"" - n + l + C' 



(14) 



The end of the proof is then similar to the content of subsection 3.3 and we obtain: 



C2 

' n+C 



< a^n < 0. 



3.5 Regularity of the modified example 

The arguments of the proof in this subsection are very similar to the ones of M. Herman. 
Let us recall that we are interested in proving that lim ||L'2(/9fc||(^o = and that 

fc— >itoo 
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lim sup{||D^(^fe||,^o; A; G Z} = . Because of the definition g, we have: H-D^V'fellco = 
\\D'^hk + D'^h^^^ — 2\\(jo. Let us introduce the notation: 

hk{x) = X + Akix) = [ {1 + iPk{t))dt. 
Jo 

Then we want to estimate the norm of: 

Ck{x) = hk{x) + ^fe^i(x) -2x = Afc(x) - Afc_i(/i~^^a;). 
We differentiate to obtain: 

DCk{x) = DAk{x) - DAk-i{hl\x)D{hl\){x) 

that is: 

DC,k{x) = ipk{x) - ipk-i{K^ix)D{h'^^i){x). 

We define then fk : [0,4] ^ [0,4] by: fkix) = hk-i{^j^x), then we have h'^^-^{x) = 
^/-^(x). We have: 

D{K\)i^) = ^J^iDf-'){x). 

'•■k 

Let us recall that: 
Therefore 

i>k-iihk\x) = Kk-iri{^) + cxk-il±{^) = Kk-ivi^) + afc-i7±(^). 



Observe that 



and then: 



D{hk\){x) = -r^ -r^ 

1 + Kk-MV) + afc_i7±( V) 



Finally, we obtain: 



V'fe-i(Vi^)^(Vi)(^) = 
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Moreover, we have: 
and 

Df^H-) = ^^K'-M = ^ 7^ 7^ 

^^-il + Kfe_ir/(^) + afc_i7±(\^) 

Let us now compute for x G T\{ (even if is two times differentiable at this point, 
the terms in the sum are not differentiable at 

D^Ckix) = DiPkix) - Di>k-i{K\x) {D{h^\){x)f - i^k-i{K^ix)D\h^\){x). 
Following [11 J, we define: 



i.e: 



and 



... , .,-1 Nn2/^-i ^r N i'k-i{K\x)DiPk-iihf^\x)D{h,^\){x) 

IVk = -i^k-l{K^lX)D (Vl)(^) = 77—^ .,-1 ^^2 

ek.,i^k-i{h,\x)D4;k-i{h,\x)D{f,')ix) 



IVk 



(i + Vfc-i(/ifcV))' 



Then 



D\k{x) = Ilk + I Ilk + IVk + Vk. 
Let us now estimate each term of this sum. We need some inequalities: 

Ci < pryllc-o, ||D7±llco, Ili^Mlco, P^Tillco < C2; (15) 
We deduce from subsections 13^ and 13^ that: 
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and therefore, we have uniformly in C >> 1 (see ([5|): 



ai. 



Urn 

fc— >itoo 



From 



l + Kk + ak + 



1 



1 + Kk-i + afc-i 



we deduce: 

|afc - Ok-i - {mk -2) + Kk- A'fc-il < C2{\Kk^i\ + |afc_i|)^ 
and by (§, (§, ([l6]): 



|afc - ak-i\ < 



Co 



Moreover, we have: 

Dfk{x) - 1 = 

and 



4 



and then we deduce from ([s]) and (16) that: 

\k\ + C' 



sup{||Z)/fc-l||co,||D/,-^-l||co}< 



(17) 



(18) 



(19) 



Let us estimate Ih = (ff - ^)^^(t) + ~ T^^^^^?^)' because of W, (|15) 
and (18), we have \IIk\ < (|fc|_^)2£^ and then, by (5), uniformly in C >> 1, we 



nave: 



lim |/4| = 

A:— >itoo 



From (3), (16) and (19), we deduce that \IIIk\ < and then uniformly in 

C » 17 we nave: 



lim = 

fc— )-±oo 



We have: 



IVk 



lu-i^k-i{K\x)Di^k-i{K\x)D{f-^){x) 



(i + V^,_i(/i^V))' 
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We deduce from (3), (15), (16) that \IVk\ < ^^^^^^^ and then that uniformly in 
C » 1, we have: 



hm \IVk\ = 

fc— >±oo 



We have for x G [0, £k]- 



then by (15) and (19): |Dr/(f ) - Dr]{ 



< 



C2 

|fc|+c- 



Because for every x G [O,^^], and — are in the same half interval of [0, 1], 7-1- is 

smooth between ^ and and be can do for 7-1- the same estimate as for r/. By (3) 
and (16), we deduce: 



\yk\< 



Co 



4(|A:| + C)2 
then, by ([s]), uniformly in C >> 1, we have: 

lim iFfcl =0. 

fc— >itoo 



Finally, we have proved that is and even that ||<^||(72 is small. 



3.6 Stable and unstable sets of the invariant curve 

We denote by T the invariant curve, that is the graph of g — Id. 

We recall that the segment with length ^ that has same center fik as Ik = [fJ-k — 
^,lik + ^] is denoted by Jk = [/^fc — /"fc + Moreover, because of the definition 
of hk and Qk (see subsection |3.1[ ), have: 

1. if /c > 1, then: 

Vx G [/ife - ^-^ , ^ik], g{x) = Hk+i + - f^k); 

o Ik 

in this case, %^ < 1; 

2. if < 0, then: 

Vx G [fJ-k,fJ'k + ^],a{x) = Hk+i + - /Ufe); 

o Ik 

if A; = then %±i < 1 and if /fc < -1 then %±i > 1. 
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We deduce that: 

1. V/c > l,g{[^J.k - = W+i - %^,^fc+i]; then 5'|[^^_ffc ^^^] is a hnear contrac- 

tion; 

4n _ [,,, , ,,, , I ^fc 



2. yk < l,g ^([/zfc,/ifc + -f]) = [nk-i,fJ'k-i + "V"] and {g is Hnear, a 

contraction if A; < and a dilatation if /c = 1. 
We introduce the family {Sk)k>i and {Uk)k<o of segments of T x M defined by: 

'S'fc = - j;); j; G [/ife - — and C/fc = {(a::, 5(2^) - a^); S [/ifc, ^fc + }• 

o o 

Because the curve F is the graph oi g — Id, these segments are subsets of F. We have: 

> I, f^{Sk) = Sk+i and ^k < 0, f-\Uk) = Uk^r, 

in the first case, f,p\s,, is a linear contraction with rapport and in the second case 
f~:lr is a linear contraction with rapport 



We have proved in subsection 3.2 some equalities for hk + ^^j'^i that implies: 
yk G Z, Vx G Jk,(p{x) = {mk - 2){x - fik) + fJ-k+i + ^fc-i - 2/ifc- 
Let us recall that: 

U{e,r) = {e + r,r + ip{e + r)) and f-\9,r) = {9 - r + ip{e),r - ip{e)) 

therefore the restriction of to any band x M is linear. If we know the expression 
of a linear map on a segment, we can deduce the expression of the map on the whole 
line supporting the segment. In particular, if we define the families of segments {Sk)k>i 
and {Uk)k<o by : 

S'fc = - /^fc + - l)(a; - /Ufc));x G Jfe} for A; > 1 (20) 



and 



f/fc = {(x,^fe+i - Aifc + (-^ - l)(a; - G Jfc} for A; < (21) 

f-k 

then we have Uk C Uk, Sk C Sk and: 

yk > I, U{Sk) = Sk+i and yk < 0, f-\Uk) = Uk-i- 

Moreover, the restriction of f^p to Sk is a linear contraction with rapport and the 
restriction of /^^ to C/^ is a linear contraction with rapport -j^- We then deduce that 
Sk is in the stable set of the point {fj-k, fJ-k+i — fJ'k) and that Uk is in the unstable set of 
the point {iik,^ik+i - fJ-k)- 

We then extend these two families of segments by: 
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if A:<0, Sk = f!:-\S_ 



• if A;>0, Uk = f!^iUo). 

Let us now choose a injective map 7^ : M — t- Si such that 7f(0) = xi 
(A'1)/^2 — /Ui), 7f(M) is Si without its ends and 7f(] — 00, 0[) is without its ends. 
Similarly, we choose a C°° injective map 79 : M — )■ [/q such that 7o (0) = xq = (/io, fJ-i 
fj-o), 7o(IS) is Uq without its ends and 7o(]0,+oo[) is Uq without its ends. 

We extend these curves to two families by: 7I = fl^~^ o 7^ and 7^ = Z,^ o 7^. Thi 
we have: 



1- U°lk= 7fc+i and UojI = 7^^^; 

2. Wy e Yoim, liin dif^y, f^xo) = and Vy G lim /^"xq) = 0; 

3. 7^(] - 00, 0]) U 7lf ([0, +oo[) C f^-\Si) U /^(f/o) C T. 

Let us now prove that 7f (]0, +00 [) U 7^(] - cx), 0[) C T x M\r. We will deduce that 
7f (]0, +00 [) is a part of the stable set of xi and of F that doesn't meet F, hence it is 
in an instability zone U and F is in the boundary of U (see the proposition contained 



in subsection 1.3); we will even see that U is under F. Similarly, we will prove that 



7o (] — 00, 0[) is in Z^. We will of course deduce that: 

Vfc G Z, 7|(] - 00, 0[) U 7^(]0, +oo[) C 



By (20), we have an explicit expression for: 



£2 h 
7f(]0,+oo[) = {(x,/i2 -m + (^^ ~ l)(x-^i));x e]^i,/ii + — [} 



Moreover, because of the definition of F, we have (see subsection 3.1): 



Vx + --r[,g{x) - X = (— + ai - l)(x - m) + /U2 - 

As ai > and F is the graph of (7 — Id, we deduce that 7f (]0, +oo[) doesn't meet F, and 
even that 7f (]0, +00 [) is under F. A similar argument gives the result for 7q (] — 00, 0[). 
Remark. 1) If we exchange 7_ and 7+, we obtain an instability zone U that is above 
F. 

2) If we use a similar construction along two wandering intervals, we obtain a curve F 
that is at the boundary of two instability zones. 
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4 The case of the topology: proof of theorem 

m 

If U is an open subset of A, the set Diff(Jj(f/) of symplectic diffeomorphisms of U is 
endowed with the strong Whitney's topology (see [12l |20j). Observe that the set T of 
symplectic twist maps of A is open for the Whitney topology in Diff^(A). Then if U 
is any open subset of A, the set T{U) of the restrictions to U of symplectic twist maps 
of A is open in Diff^(C/). 
The following result is theorem 3 of [2]. 

Theorem. ([2j) Let (M, w) be a non-compact closed manifold. There exists a dense 
Gs subset Q o/Diff;^(M) such that, for all f & Q, the set of points of M whose positive 
orbit is relatively compact in M has no interior. 

Let us now consider an essential invariant curve T of a symplectic twist map / of 
A. The curve F is then the graph of a Lipschitz map 7 : T — t- M. Denoting by U one 
of the two connected components of A\r, we have: f{U) = U. In order to define a 
neighborhood U of f\u for the strong topology, we use the function £:[/—)• 
defined by e{9, r) = {r — j{0))'^. 

U = {ge T{Uy,y{e,r) e U,snp{d{f{9,r),g{d,r)), \\Df{9,r)-Dg{e,r)\\} < e{9,r)}. 

The previous theorem implies that there exists h £V( such that the set of points of U 
whose positive orbit for h is relatively compact in U has no interior. 

We now define g : A ^ A such that g\j^\u = f\A\u ^^"^ 9\u — ^- I* comes from the 
definition oiU and T{U) that g £ Diffi(A). 

Let us prove that U contains at most one essential invariant curve for g. If not, there 
exists a bounded invariant open region R between two such invariant curves. Then all 
the points of R have a positive orbit that is relatively compact in U, this is a contra- 
diction with the choice of h. 

Let us now assume that /|r has an irrational rotation number or that /|r is conju- 



gated to a rational rotation. We have noticed in subsection 1.3 that in this case, if T* 
is another essential invariant curve of /, than F n L* = 0. Hence the closure U ol U 
contains at most one essential invariant curve that is diff'erent from F, and this curve 
is contained in U . There are two cases: 

• either U contains one essential invariant curve F' for g. The region R between F 
and F' is an instability zone for g and its boundary contains F; 

• or [/ contains no essential invariant curve for g. The curve F is at the boundary 
of the instability zone U of g. 
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Remark. Using methods contained in the (non-published) thesis of my student Marie 
Girard that allows us to destroy all the invariant curves by perturbation, we can choose 
h such that U contains no essential invariant curve and thus is an instability zone. 

The only case that we did not solve is the case of a rational rotation number for 
/|r and a hyperbolic dynamics for /|p (i.e. we assume that all the periodic points of 
/|r are hyperbolic). 

C. Robinson proved the following result in [21J. Let us recall that a periodic point p 
of / with period r is non-degenerate if no root of 1 is an eigenvalue of Df'^{p). 

Theorem. ( |21j ) Let {M,uj) be a closed manifold. There exists a dense Gg subset Q of 
Diff^(M) such that, for all f & G, the periodic points are non degenerate and the stable 
and unstable manifolds of each pair of hyperbolic periodic orbits of f are transverse at 
all of their points of intersection. 

We assume now that / G T{^) a symplectic twist map that has an essential invari- 
ant curve r such that: 

• its rotation number is rational; 

• the periodic points of /|r are all hyperbolic. 

Then there is a finite number of such periodic points, that we denote by xi, . . . 
and r is the union of {xi, . . . , x„} and some branches of the stable/unstable manifolds 
of these periodic points. Let us notice that every g £ U can be extend in a unique 
g E T(A) by: g\A\u = /|A\w ^-^d that in this case, -D^ir = Df^j^. Hence g has the same 
periodic points as / on T, and this periodic points are hyperbolic. 

We can directly adapt Robinson's proof to build a dense Gs G oiU such that for 
all 5 € t/, the stable and unstable branches of the stable and unstable manifolds of the 
Xi for g that are contained in U are transverse at all of their points of intersection. 

If now r* is an essential invariant curve for g that is contained in U and that meets 
r, then mr* is a closed invariant set that contains a point of the stable manifold of a 
point Xi. Hence it contains this Xj. The rotation number of F* is then equal to the one 
of r, and then T* is the union of {xi, . . . , x„} and some branches of the stable/unstable 
manifolds of these periodic points. But if P 7^ P*, then P* contains a branch in U that 
is a stable and an unstable branch, and this contradicts the transversality of such 
branches. We deduce that either P = P* or P n P* =0, and we can conclude exactly 
in the same way as is the irrational case. 
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